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Abstract 



The growing number of dimensionality reduction (DR) methods available for data vi- 
sualization has recently inspired the development of quality assessment measures, in order 
to evaluate the resulting low-dimensional representation independently from a methods' 
inherent criteria. Several (existing) quality measures can be (re)formulated based on the 
, so-called co-ranking matrix, which subsumes all rank errors (i.e., differences between the 

(^^^ ' ranking of distances from every point to all others, comparing the low-dimensional rep- 

resentation to the original data). The measures are often based on the partioning of the 
co-ranking matrix into 4 submatrices, divided at the K-th row and K-th column, calculat- 
ing a weighted combination of the sums of each submatrix. Hence, the evaluation process 
typically involves plotting a graph over several (or even all possible) settings of the param- 
eter K. Considering simple artificial examples, we argue that this parameter controls two 
notions at once, that need not necessarily be combined, and that the rectangular shape 
, of submatrices is disadvantageous for an intuitive interpretation of the parameter. We de- 

}_( ' bate that quality measures, as general and flexible evaluation tools, should have parameters 

I with a direct and intuitive interpretation as to which specific error types are tolerated or 

penalized. Therefore, we propose to replace the parameter K with two distinct parameters 
to control these notions separately, and introduce a differently shaped weighting scheme 
on the co-ranking matrix. The two new parameters can then directly be interpreted, re- 
spectively, as a threshold up to which rank errors are tolerated, and a threshold up to 
which the rank-distances are significant for the quality evaluation. Moreover, we propose a 
color representation of local quality to visually support the evaluation process for a given 
mapping, where every point in the mapping is colored according to its local contribution 
to the overall quality value. 
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1 Introduction 



The amount of electronic data available today is becoming larger and larger in virtually all 
application domains; at the same time, its complexity and dimensionality is increasing rapidly 
due to improved sensor technology and fine grained measurements as well as dedicated data 
formats. In consequence, humans can no longer directly deal with such collections by inspecting 
the text files. Rather, automated tools are required to support humans to extract the relevant 
information. One core technology is given by data visualization: relying on one of the most 
powerful senses of humans, it offers the possibility to visually inspect large amounts of data at 
once and to infer relevant information based on the astonishing cognitive capabilities of humans 
in visual grouping and similar. 

Dimensionality reduction techniques constitute one important method in understanding 
high-dimensional data because they directly produce a low-dimensional visualization from high- 
dimensional vectorial data. Consequently, many dimensionality reduction techniques have been 
proposed in recent years. In the beginning, these methods were primarily linear, like principal 
component analysis (PC A), corresponding to low cost dimensionality reduction techniques with 
a well founded mathematical background. However, linear techniques cannot preserve relevant 
nonlinear structural elements of data. Therefore, recently, more and more nonlinear methods 
like Isomap [I], locally linear embedding (LLE) ^ , and stochastic neighbor embedding (SNE) [3] 
have become popular, see the overview [4j, for example. 

With more and more dimensionality reduction techniques being readily available, the user 
faces the problem which of the methods to choose for the current application. Usually, different 
techniques can lead to qualitatively very different results. This is due to several reasons: From a 
theoretical point of view, dimensionality reduction (DR) constitutes an ill-posed problem: Not 
all relations that exist in the data in the high-dimensional space can be faithfully represented 
in the low-dimensional space, and it is not clear, which relations should be preserved. This 
has resulted in different objective functions for the various DR techniques corresponding to 
qualitatively very different visualizations for a given data set. In addition, virtually all recent 
techniques have parameters to control the mapping. These parameters usually control (in some 
way) which relations should be preserved best. Hence, depending on the parameters of the 
method, even a single DR technique can lead to qualitatively very diverse results. In addition, 
many techniques do not lead to a unique solution, rather, multiple different outputs can be 
reached due to random aspects of the algorithm corresponding to different local optima of the 
objective. Thus, it is even possible that qualitatively different results can be obtained by a 
single method with a single set of model parameters. 

It is usually not clear whether the different results obtained by one or several DR techniques 
correspond to different relevant structural aspects in the data which are possibly partially 
contradictory in low dimensions, or whether some of the methods and model parameters are 
less suited to preserve the relevant structural aspects in the given data set. Further, it can 
happen that suboptimal results are obtained simply because of numerical problems of a given 
technique such as (bad) local optima. At the same time, it is very hard for humans to judge the 
quality of a given mapping and the suitability of a specific technique and choice of parameters by 
visual inspection: the original data set is not accessible to the user due to its high dimensionality 
such that a human cannot compare a given visualization to ground truth easily. Therefore, there 
is a need to develop formal measures which judge the quality of a given mapping of data. Such 
formal measures should evaluate, in an automated and objective way, in how far structure of 
the original data corresponds to the structure observed in the low-dimensional representation. 

Formal measures to evaluate the quality of a data visualization have several benefits: a 
formal measure directly indicates whether an observer can trust a given mapping or whether 
effects of the data visualization are caused by the choice of the method and parameters rather 
than structures in the data set. Further, a formal quality measure can help humans to choose 
a good method and a good set of parameters for a given data set. Often, it is not easy to 
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predict the effects of model parameters on the resuhing mapping. Good quahty measures can 
serve as a means to interactively optimize the parameters. Last not least, quality measures are 
generally important to automatically evaluate and compare DR techniques for research. 

Several quality criteria to evaluate dimensionality reduction have been proposed in recent 
years, see [5] for an overview. As for dimensionality reduction itself, the problem to define 
formal evaluation criteria suffers from the ill-posedness of the task: it is not clear a priori which 
structural aspects of the data should be preserved in a given task. Most quality measures which 
have been proposed recently are based on neighborhood relations of the data. They measure 
in some way in how far neighborhood relationships (e.g. ranks of data points) correspond to 
each other in the original space compared to the projection. Two recent quality measures 
offer a general approach and constitute frameworks that include earlier measures as special 
cases [51 [S]. Regarding this general framework, it becomes apparent that also the quality 
measure eventually depends on the needs of the user since the user can specify, depending on 
the task, which aspects of the data are particularly relevant. For example, when dealing with 
medical data, the user would prefer false positives over false negatives (whatever those may be 
in the case of dimensionality reduction). Also, data in which manifold structures are expected 
should be evaluated differently than clustered data. 

Therefore, there is a need for intuitive and easily accessible quality measures which allow 
the user to determine the precise form of the measure based on the current application. That 
means, there is a need for quality measures with parameters which have an intuitive meaning 
and which can easily be set by the user. The co-ranking matrix 5 already goes into this 
direction by pointing out the relevance of the neighborhood rank which the user believes is 
important. 

We will discuss that the global quality measure which has been derived based on this frame- 
work in the work [S] does not correspond to an intuitive interpretation by the user: on the one 
hand, it depends on absolute values of the ranks rather than the deviation of the ranks, i.e. the 
actual 'errors' made by a DR method. On the other hand, it relies on a single parameter only, 
the size of ranks taken into account, which controls both aspects: which errors are tolerated 
and which neighborhood relations are considered interesting for the mapping. We show in a 
simple example that this quality measure leads to unexpected values which do not correspond 
to an intuitive understanding. 

As an alternative, based on the co-ranking framework, we propose a different family of 
quality criteria which are based on the values of the rank errors rather than the absolute values 
of the ranks. This family is parameterized by two parameters which control the size of errors 
which are tolerated on the one hand and the size of the neighborhood of points which should 
be mapped faithfully by the dimensionality reduction on the other hand. This way, the user 
can intuitively control the resulting quality measure. We also propose an intuitive way to link 
formal quality criteria to a given visualization and to actually visualize the quality of a mapping 
such that the user can immediately see which parts of the mapping are trustworthy and which 
are not. 

2 Dimensionality Reduction and Quality Measures 

Dimensionality reduction techniques are used for visualization by mapping a high-dimensional 
dataset S = {^i, . . . ,^n} to a low-dimensional dataset X — {xi, . . . ,X]\[}. By design and via 
parameters, DR methods specify which properties should be maintained by the mapping. Some 
techniques are based on global mappings such as linear techniques, which determine a matrix 
to reduce the dimensionality of the data set by a linear transformation, or topographic mapping 
such as the self-organizing map 7:, which parameterize a mapping by a lattice of prototypes 
in the data space. Many modern nonlinear techniques are non-parametric: they map a given 
set of data points directly to their respective projections without specifying a functional form. 
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This way, the mapping has large flexibihty and highly nonlinear effects can be obtained. 

Non-parametric dimensionality reduction is often based on a cost function or objective, 
which evaluates in how far characteristics of the original data are preserved by the projections 
Xi. Appropriate projections are then determined minimizing this objective with respect to the 
parameters Xi. For example, t-SNE maintains the neighborhood probabilities in both spaces, 
while LLE tries to place points in such a way that locally linear neighborhoods are maintained. 
See e.g. the article [5] for a general formalization of popular non-parametric dimensionality 
reduction techniques in this way. 

Let us look more closely at the LLE algorithm. First, one fixes a parameter K, that 
specifies the number of neighbors used to reconstruct a point. For each point ^i, find the K 
closest neighbors and find weights Wij such that is best represented by X^jLi ''^ij^jj where 
at most K of the w^j's are allowed to be non-zero. Finally, low-dimensional mapped points Xi 
are chosen such that the differences between Xi and its neighbor- reconstruction ^ • _-i W{j Xj are 
minimal. 

A similar process description can be given for t-SNE J^. First, using a fixed perplexity, 
the probabilities Tr^j that point i in the high-dimensional space is neighbored to point j in 
the high-dimensional space are estimated. Similar probabilities pij for the low-dimensional 
space are estimated using a Student-t distribution. The positions in the low-dimensional space 
are optimized by minimizing the KuUback-Leibler divergence between these distributions, i.e., 
minimizing their dissimilarity. 

In both cases, a quality criterion, depending on a parameter, is used to specify an optimal 
mapping. Thus, for non-parametric DR methods, there is often a close relationship of an 
objective function which in some way evaluates the quality of a mapping, and a DR algorithm 
which actually finds projections such that the quality is optimized. Of course, these criteria 
arc partially chosen because they have a nice corresponding optimization method. For LLE, 
for example, a convex objective guarantees unique solutions. However, as demonstrated in the 
article [B] , a quality criterion itself can also be used to derive a DR model by means of a numeric 
optimization technique. Thus, a quality criterion gives rise to a corresponding optimal mapping. 
Conversely, most existing non-parametric DR methods induce a quality measure by which a 
DR mapping can be evaluated. Naturally, these evaluation criteria are highly biased towards 
the corresponding DR technique. Further, they are usually not parameterized in an intuitive 
way and they also incorporate aspects which are important in order to obtain a numerically 
appealing optimization method rather than a valid evaluation criterion only. 

Here we are interested in a criterion which evaluates the quality of DR mappings in a 
uniform and intuitive way, and which provides a parameterization that allows for an intuitive 
control by the user. Thereby, it is irrelevant whether the resulting objective also leads to a 
simple optimization scheme. First aproaches in this direction have been proposed based on the 
co-ranking framework in the articles [51 [S] . 

3 The Co-ranking Framework 

In the following, we introduce the co-ranking framework as proposed by Lee and Verleysen [S] 
and we recall several established quality criteria based thereon. Let 5ij be the distance from 
to in the high-dimensional space. Analogously, dij is the distance from Xi to Xj in the 
low-dimensional space. From these distances we can compute the ranks of the neighbors for 
each point. The rank of with respect to in the high-dimensional space is given by 

Pij = \{k I dik < Sij or {Sik = Sij and I < k < j < N)}\, 

where \A\ is the cardinality of the set A, resulting in the ranks {1, . . . , — 1}. Analogously, 
the rank of Xj with respect to Xi in the low-dimensional space is given by 

Tij = |{fc I dik < dij or (dik = dij and 1 < fc < j < A^)}]. 
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Figure 1: Large-scale structure of the co-ranking matrix. On the left, the matrix is split into 
blocks to show different types of intrusions and extrusions. In a perfect mapping, the co-ranking 
matrix will be a diagonal matrix. The image on the right shows how rank differences will alter 
the matrix. If a neighbor moves further away in the low-dimensional space, an extrusion, it 
will move mass to the right of the diagonal. Similarly, intrusions move mass to the left of the 
diagonal. 



Many existing quality criteria measure in how far ranks of points are preserved after the reduc- 
tion to a low-dimensional space. This way, local relationships are evaluated without referring 
to irrelevant issues such as e.g. scaling of the data. 

To generalize such measures, the co-ranking matrix Q [10 is defined by 

Qki = |{(«, j) I Pij = k and = ;}|. 

Errors of a DR mapping correspond to off-diagonal entries of this co-ranking matrix. A point 
i that gets a lower rank with respect to a point i in the low-dimensional space than in the 
high-dimensional space, i.e. pij > rij, is called an intrusion. Analogously, if has a higher 
rank in the low-dimensional space it is called an extrusion. As shown in Figure [TJ intrusions 
and extrusions correspond to off-diagonal entries in the upper or lower triangle, respectively. 

Usually, a DR mapping is not used to map all relationships of data faithfully. Rather, the 
preservation of local relationships is important. Hence, rank errors for large ranks are not 
as critical as rank errors of close points. For this reason, Lee and Verleyssen distinguish two 
types of intrusions/extrusions, those within a if -neighborhood, which are benevolent, and those 
moving across this boundary, which are malign with respect to quality. A iiT-intrusion (resp. K- 
extrusion) is an intrusion for which r.y < K (resp. pij < K). Subsequently, mild if- intrusions 
are events for which < < K, while hard if- intrusions are defined by < K < pij. Mild 
if-extrusions and hard if-extrusions are defined accordingly. 

Based on this setting, a simple quality measure can be defined: it counts the number of 
points that remain inside the if -neighborhood while projecting, i.e., all points which keep their 
rank, and all mild in- and extrusions: 

K K 

Q^^iK) = — Y^Y^^ki. (1) 

fe=i 1=1 

The normalization ensures that the quality of a perfect mapping equals oneQ 

The quality criterion is very similar to the local continuity meta-criterion (LCMC) that was 
proposed by Chen and Buja In fact, it coincides up to a linear term that accounts for the 

^Instead of expressing the quality, one could define a measure of error analogously, as 1 — QnxC^)- 
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quality of a random mapping: 



LCMC(if) = QMK) - (2) 

Note that the range of this quahty measure depends on K, i.e. the size of the neighborhood 
which should be preserved by a DR mapping. Often, a graph of the quality values over all 
possible K (or a sufhcient selection thereof) is plotted. These measures have the disadvantage 
that they depend on if, and thus do not give a single decisive number that determines the 
quality of the mapping. Following the idea of locality-parameters in DR techniques (e.g., the 
number of neighbors in LLE and the perplexity in t-SNE) Lee and Verleysen make a similar 
local vs. global evaluation of the quality graph. 

To estimate which values of K should be considered local, the following splitting point was 
proposed in [5]: 

/ K 
Kma^ = argnmxLCMC(ii') = argmax ( Qnx{K) - ^ _ ^ 

Subtracting the baseline ensures that there is a well-defined maximum that favors locality. 
Given the splitting point, a local and a global quality measure is obtained by averaging the 
respective parts of the quality graphs: 

Qlocal = ^ Qnx{K), (3) 



K 



AT-l 



Qglobal = ^TT^^^^ ^ QMK). (4) 



K=K 



Both values range from to 1. As local properties are more important, the authors advise to 
rank methods according to Qiocai and only to consider Qgiobai in case of a tie. 



4 A family of quality criteria based on rank errors 

The co-ranking matrix as introduced in [5j offers a very elegant framework to formalize quality 
criteria based on rank errors. However, it has a severe drawback: The quality ([IJ) depends on 
the number of rank errors in a region of interest only, disregarding the size of rank errors. 

Let us have a look at the evaluation measure ([T]). A region of interest, i.e. a rank K is 
fixed, following the idea that ranks which are very large (larger than K) are not meaningful in 
the data space and the projection space and thus, they can be disregarded. Hence, only errors 
within the region of interest are counted. The second role of K is to define what is regarded an 
error: an error occurs if and only if the region of interest in the original space and the projection 
space does not coincide. Hence, the actual size of the rank error is not important. Rather, it is 
checked whether ranks < K keep this property while projecting and vice versa. As an extreme 
case, points which change their rank from 1 to K are not counted as errors, while points which 
change their rank only from K to K + 1 are. 

This definition of Q^x{K) can lead to curious situations, which demonstrate the unintuitive 
character of the parameter K. Consider the pairwise swapping of points that is shown in 
Figure [51 The number of points can be chosen arbitrarily. Examining the structure quickly 
shows that the maximum rank error between these permutations is at most 4 (for example, 
when the base point moves left, and the other point moves rig htB Intuitively, if we consider 

^Note that the points are equidistant, and in case of a tie in distances, the point with the lower alphabetical 
letter gets assigned the lower rank. 
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rank error sizes up to 4 as acceptable, this mapping is perfect. This is, however, not the case 
when looking at (5nx(5): there are still errors. In fact, for every value of K there will be some 
point that moves from, for example, rank K to a slightly higher rank, and therefore be counted 



as an error. This is also confirmed by the graph in Figure 3(a) which displays the quality for a 
row of 20 points. It is hardly possible to detect an intuitive correlation of the size of QNx(-fi') 
and the parameter K based on the observation of the mapping, albeit its errors can be fully 
characterized by local pairwise swappings. 

A look at the co-ranking matrix in Fig. [5] indicates the underlying structure in this case. 
Since the rank error is always smaller than 5, only 4 off-diagonals of the co-ranking matrix 
are non- vanishing. Note that the zth off-diagonal corresponds to rank errors of size i. The 
quality measure (jlj, however, only sums over rectangular parts of the co-ranking matrix. This 
observation also suggests how the quality ([T|) can be altered to lead to a more intuitive pa- 
rameterization: rather than rectangular parts only, it should focus on a limited number of 
off-diagonals corresponding to the size of the rank deviation which is considered acceptable. 

As already mentioned, K has also a different role by singling out the region of interest 
based on the rationale that large ranks are not relevant anyway. For the original quality ([1]), 
this choice is controlled by the same parameter K: points up to rank K only are considered 
relevant, at the same time leading to the consequence that errors within this range K are fully 
accepted. It seems more intuitive to separating this control parameter: in addition to the size 
of the errors which are tolerated, a region of interest should be controlled independently. 

Now, we will formalize this consideration by first reformulating the quality ([1]) such that the 
two different roles of K become apparent, and then generalizing this formalization such that 
an explicit control of the region of interest and the tolerated rank error size becomes possible. 
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Figure 2: On the left is an example mapping from a one-dimensional set of equidistant points to 
a slight reordering. Since the points are only pairwise swapped, the changes in rank-distances 
are rather small. For the same setup with 20 points, this is confirmed by the co-ranking matrix 
that is shown on the right, with entries depicted on a gray scale. White indicates a zero value, 
while black corresponds to the maximum value in the matrix. 



4.1 Analysis of (5nx(^) 

As discussed in formal evaluation measures judge in how far ranks of points are preserved. 
Hence, the evaluation measures are based on rank errors 

El J = \Pij-rij\ 

which measure the deviation of the rank of two points in the original space and the projection 
space, respectively. The parameter K has two distinct functions: it limits the region of interest, 
i.e. the relevant ranks which are regarded when evaluating the quality; and it specifies what is 
regarded as an error, i.e. which rank deviations are tolerated. 

Formally, the first role of K can be captured by a rank significance function : -R x i? x N — > 
[0, 1] that determines, for any pair of points i and j the extent Ws{pij,rij, K) to which their 
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(a) Quality QNx(^f) (b) Quality Qj^^C'^s , tt) 



Figure 3: The figure shows quality evaluations for the mapping presented in Figure [H on 
the left with the established measure Qj^x{K), on the right with the proposed new measure 
Qnx('*si '^t)- Both are evaluated for all possible parameter settings K and (ks, Kt) respectively. 
The particular position of if = 5 is highlighted on the graph in the left figure. Qnx{K) with 
K > 5 does not yield values of 1 entirely, which seems rather unintuitive for the given mapping. 
As expected, the matrix for Qnx('*'si ^t) does have ones, for all > 5. 



rank error should be taken into account. For the quality measure Q]S!x{K), we have: 



pij > K A rij > K 
otherwise. 



To describe the second role of we use a function wt : Rx RxN [0, 1] that determines 
the weight of the rank error Eij for points i and j based on their ranks pij and . According 
to the definition of Q-MxiK), we have the following error tolerance function: 



Wt{pij,rij,K) 



{p^J <K A r,j > K)\J {p,j > KA < K) 

1 otherwise. 



where we count the overlap of the K neighborhoods in the original space and the projection 
space, respectively. The quality is proportional to the number of points which are in the region 
of interest and which error is acceptable according to the error tolerance: 



^ N N 

1=1 j=i 

As discussed before, a problem of the error tolerance function wt is that this function 
depends on the actual ranks and not on the rank error. Examining Figure [1] confirms this. A 
point with high-dimensional rank 1 and low-dimensional rank K is acceptable, although it has 
an absolute rank error oi K — 1. On the other hand, a point that has high-dimensional rank K 
and low-dimensional rank K + 1 is not acceptable, although its rank error is only 1. 

4.2 A quality measure based on rank errors 

Because of this fact, we propose the following alternative error tolerance function 

/ \ / \pij - Tijl > Kt 

Wtip..„r,„Kt)^^ 1 otherwise, 

that depends on the rank error rather than the value of the ranks. The cut-off value Kt deter- 
mines which error sizes are accepted. We use the same rank significance function Ws as in ([S]), 
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Figure 4: New weighting scheme for the co-ranking matrix (right) as compared to the one 
proposed by Lee and Verleysen [5] (left): as before, the region of interest is hmited to points 
which have a rank smaher than Ks either in the original space or the projection space, i.e. the 
lower right corner is not considered. Within this region of interest, all pairs of points with a 
rank error smaller than a given tolerance Kt are counted as correctly projected. 



but substitute the parameter K by the the cut-off parameter k^. Following equation ([S]), we 
then get a new quality measure: 

^ N N 

<9Nx('«s>'«t) = — ^y"y!";^(Pu>''y:«^^) ■Wt{pij,nj,Kt). (6) 

1=1 j=l 

Because of the normalization, values are in the interval [0, 1] with 1 corresponding to a perfect 
mapping. Figure |4] shows the region of the co-ranking matrix which is taken into account in 
this quality measure. One might also consider more complex or smooth functions for Wg and 
Wt than simple cut-offs with Hs and nt respectively and corresponding normalization factors, 
to account for a smooth weighting of the neighborhood size and the size of the errors.. 

The new quality measure Q'jqxi'^s, Kt) now depends on two parameters instead of only one 
K which allow for an intuitive access: determines which size of the rank errors are tolerated, 
while Ks singles out which ranks fall into the region of interest. 

We can no longer plot only one graph to picture the quality of a mapping at different stages 
of K. To get a rich impression of a mapping's qualitative characteristics, one can instead display 
a 3D surface or colored matrix, where the position {Ks,Kt) is assigned the value Q'^-^Iks, Kt)^ 
see Figure [3(b)| for an example. The matrix in Figure [3 (b) | shows all values of Q'j^y^{Ks , nt) for 
the example in Figure [2l It clearly shows that the maximum quality is reached for all Ks > 4. 

Similar to [S], it is possible to subtract the baseline given by a random mapping. Based 
on this 'centered' graph, a baseline can be determined up to which border local values in the 
matrix are interesting. Then, the given matrix can be connected to a single scalar by averaging 
over these numbers. 



5 Local quality Assessment 

The quality criteria introduced in the previous section average the contributions of all points. 
It can be useful to visually represent the quality (or analogously error) of a single point, in 
order to gain insight into local qualitative changes, especially when the deviation of the quality 
of the mapping in the single parts is very high. This principle has been used, for example, to 
visualize the topographic distortion of self-organizing maps, where one can display the distance 
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(b) High- and low-dimensional ranks 



Figure 5: The items a and b are interchanged in the one-dimensional dataset shown on the left. 
This resuhs in the rank-matrices as shown on the right, errors are highhghted. 



between neurons in the data space as a color in the topographic map, see [7]. Similarly, in the 
approach [T^, the local topographic reliability of dimensionality reduction is displayed. 

The quality measure as introduced above naturally gives rise to a local quality which, given 
a single point, displays the trustworthiness of the map in this area. 

First, we examine what to count as visualization errors. To ease this, we assume that every 
rank change is counted as an error for now, i.e. = 1 and Ks = iV — 1. As a first attempt, one 
might choose the following point-wise error measure for point i: 

1 ^ 

Q't = 77 Y] Ws [pij ,rij,K,)-Wt (Py , nj ,Kt). 

However, this is not optimal. Consider the simple example shown in Figure [S] Interchanging 
the elements a and c induces only rank changes for point b. Given the formula Q^, only point 
b would be indicated as erroneous, which does not really reflect our intuitive notion of error. 
Similarly, we can assume fixed positions for a and c, and move b towards c. Then the ranks for 
a and c change, while b's remain unchanged, and hence b gets no error. Therefore, we propose 
to use the symmetric quality: 

N 



Q't = ^ (ws {Ptj , nj ,Ks)-Wt {prj , nj ,Kt) + (7) 

■Ws{pji,rj,,Ks) ■ Wt{pji,rji,Kt)y 



i=i 



This choice assigns a reduced quality to all the three points in the example of Figure [5) Given 
this measurement, it is easy to color every point in a low-dimensional data visualization ac- 
cording to the local quality measured at the point. 

We show an example where this simple coloring scheme clearly indicates which regions of 



a mapping can be considered trustworthy. In Figure 6(a) we show the popular 'swiss roll' 
benchmark data set, which consists of points in M'^, uniformly sampled on a spiral-shaped 
two-dimensional manifold. The data is mapped by t-SNE using a high perplexity parameter 
which produces an 'unfolded' view of the manifold, with some local tearing and distortion, see 
Figure |6(b)[ The coloring clearly reveals the tears within the manifold as well as the larger 
rank errors that occur at the rightmost points caused by 'unrolling' and putting the inner end 
of the belt far away from its original neighbors on the next spiral loop level. In a real world 
scenario, where the original data is high-dimensional and its detailed structure is unknown to 
the user, the coloring of the mapped points may help to understand local characteristics of the 
mapping. 

References 
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Figure 7: The figure shows quality evaluations for the mapping presented in Figure El on 
top with the established measure Qt^xiK)^ on the bottom with the proposed new measure 
Q'^xi'^s, Kt). Both are evaluated for all possible parameter settings K and {ks, Kt) respectively. 
We recommend to read the bottom matrix by first focusing on some error tolerance value kj, 
and then read one row of the matrix with all rank significance values Ks ■ 
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